Smooth pursuit inhibition reveals audiovisual enhancement of fast movement control.

The sudden onset of a visual object or event elicits an inhibition of eye movements at latencies approaching the minimum delay of visuomotor conductance in the brain. Typically, information presented via multiple sensory modalities, such as sound and vision, evokes stronger and more robust responses than unisensory information. Whether and how multisensory information affects ultra-short latency oculomotor inhibition is unknown. In two experiments, we investigate smooth pursuit and saccadic inhibition in response to multisensory distractors. Observers tracked a horizontally moving dot and were interrupted by an unpredictable visual, auditory, or audiovisual distractor. Distractors elicited a transient inhibition of pursuit eye velocity and catch-up saccade rate within ∼100 ms of their onset. Audiovisual distractors evoked stronger oculomotor inhibition than visual- or auditory-only distractors, indicating multisensory response enhancement. Multisensory response enhancement magnitudes were equal to the linear sum of responses to component stimuli. These results demonstrate that multisensory information affects eye movements even at ultra-short latencies, establishing a lower time boundary for multisensory-guided behavior. We conclude that oculomotor circuits must have privileged access to sensory information from multiple modalities, presumably via a fast, subcortical pathway.

Prior visual experience increases children's use of effective haptic exploration strategies in audio-tactile sound-shape correspondences.

Sound-shape correspondence refers to the preferential mapping of information across the senses, such as associating a nonsense word like bouba with rounded abstract shapes and kiki with spiky abstract shapes. Here we focused on audio-tactile (AT) sound-shape correspondences between nonsense words and abstract shapes that are felt but not seen. Despite previous research indicating a role for visual experience in establishing AT associations, it remains unclear how visual experience facilitates AT correspondences. Here we investigated one hypothesis: seeing the abstract shapes improve haptic exploration by (a) increasing effective haptic strategies and/or (b) decreasing ineffective haptic strategies. We analyzed five haptic strategies in video-recordings of 6- to 8-year-old children obtained in a previous study. We found the dominant strategy used to explore shapes differed based on visual experience. Effective strategies, which provide information about shape, were dominant in participants with prior visual experience, whereas ineffective strategies, which do not provide information about shape, were dominant in participants without prior visual experience. With prior visual experience, poking-an effective and efficient strategy-was dominant, whereas without prior visual experience, uncategorizable and ineffective strategies were dominant. These findings suggest that prior visual experience of abstract shapes in 6- to 8-year-olds can increase the effectiveness and efficiency of haptic exploration, potentially explaining why prior visual experience can increase the strength of AT sound-shape correspondences.

Body Pitch Together With Translational Body Motion Biases the Subjective Haptic Vertical.

Accurate perception of verticality is critical for postural maintenance and successful physical interaction with the world. Although previous research has examined the independent influences of body orientation and self-motion under well-controlled laboratory conditions, these factors are constantly changing and interacting in the real world. In this study, we examine the subjective haptic vertical in a real-world scenario. Here, we report a bias of verticality perception in a field experiment on the Hong Kong Peak Tram as participants traveled on a slope ranging from 6° to 26°. Mean subjective haptic vertical (SHV) increased with slope by as much as 15°, regardless of whether the eyes were open (Experiment 1) or closed (Experiment 2). Shifting the body pitch by a fixed degree in an effort to compensate for the mountain slope failed to reduce the verticality bias (Experiment 3). These manipulations separately rule out visual and vestibular inputs about absolute body pitch as contributors to our observed bias. Observations collected on a tram traveling on level ground (Experiment 4A) or in a static dental chair with a range of inclinations similar to those encountered on the mountain tram (Experiment 4B) showed no significant deviation of the subjective vertical from gravity. We conclude that the SHV error is due to a combination of large, dynamic body pitch and translational motion. These observations made in a real-world scenario represent an incentive to neuroscientists and aviation experts alike for studying perceived verticality under field conditions and raising awareness of dangerous misperceptions of verticality when body pitch and translational self-motion come together.

Collinear search impairment is luminance contrast invariant.

Collinear search impairment (CSI) is a phenomenon where a task-irrelevant collinear structure impairs a target search in a visual display. It has been suggested that CSI is monocular, occurs without the participants' access to consciousness and is possibly processed at an early visual site (e.g. V1). This effect has frequently been compared with a well-documented opposite effect called attentional capture (AC), in which salient and task-irrelevant basic features (e.g. color, orientation) enhance target detection. However, whether this phenomenon can be attributed to non-attentional factors such as collinear facilitation (CF) has not yet been formally tested. Here we used one well-established property of CF, i.e. that target contrast modulates its effect direction (facilitation vs suppression), to examine whether CSI shared similar signature profiles along different contrast levels. In other words, we tested whether CSI previously observed at the supra-threshold level was reduced or reversed at near-threshold contrast levels. Our results showed that, regardless of the luminance contrast levels, participants spent a longer time searching for targets displayed on the salient singleton collinear structure than those displayed off the structure. Contrast invariance suggests that it is unlikely that CSI is exclusively sub-served by an early vision mechanism (e.g. CF).

The feeling of "kiki": Comparing developmental changes in sound-shape correspondence for audio-visual and audio-tactile stimuli.

Sound-shape crossmodal correspondence, the naturally occurring associations between abstract visual shapes and nonsense sounds, is one aspect of multisensory processing that strengthens across early childhood. Little is known regarding whether school-aged children exhibit other variants of sound-shape correspondences such as audio-tactile (AT) associations between tactile shapes and nonsense sounds. Based on previous research in blind individuals suggesting the role of visual experience in establishing sound-shape correspondence, we hypothesized that children would show weaker AT association than adults and that children's AT association would be enhanced with visual experience of the shapes. In Experiment 1, we showed that, when asked to match shapes explored haptically via touch to nonsense words, 6- to 8-year-olds exhibited inconsistent AT associations, whereas older children and adults exhibited the expected AT associations, despite robust audio-visual (AV) associations found across all age groups in a related study. In Experiment 2, we confirmed the role of visual experience in enhancing AT association; here, 6- to 8-year-olds could exhibit the expected AT association if first exposed to the AV condition, whereas adults showed the expected AT association irrespective of whether the AV condition was tested first or second. Our finding suggests that AT sound-shape correspondence is weak early in development relative to AV sound-shape correspondence, paralleling previous findings on the development of other types of multisensory associations. The potential role of visual experience in the development of sound-shape correspondences in other senses is discussed.

Look where you go: Characterizing eye movements toward optic flow.

When we move through our environment, objects in the visual scene create optic flow patterns on the retina. Even though optic flow is ubiquitous in everyday life, it is not well understood how our eyes naturally respond to it. In small groups of human and non-human primates, optic flow triggers intuitive, uninstructed eye movements to the focus of expansion of the pattern (Knöll, Pillow, & Huk, 2018). Here, we investigate whether such intuitive oculomotor responses to optic flow are generalizable to a larger group of human observers and how eye movements are affected by motion signal strength and task instructions. Observers (N = 43) viewed expanding or contracting optic flow constructed by a cloud of moving dots radiating from or converging toward a focus of expansion that could randomly shift. Results show that 84% of observers tracked the focus of expansion with their eyes without being explicitly instructed to track. Intuitive tracking was tuned to motion signal strength: Saccades landed closer to the focus of expansion, and smooth tracking was more accurate when dot contrast, motion coherence, and translational speed were high. Under explicit tracking instruction, the eyes aligned with the focus of expansion more closely than without instruction. Our results highlight the sensitivity of intuitive eye movements as indicators of visual motion processing in dynamic contexts.

Individual Differences in Multisensory Interactions:The Influence of Temporal Phase Coherence and Auditory Salience on Visual Contrast Sensitivity.

: While previous research has investigated key factors contributing to multisensory integration in isolation, relatively little is known regarding how these factors interact, especially when considering the enhancement of visual contrast sensitivity by a task-irrelevant sound. Here we explored how auditory stimulus properties, namely salience and temporal phase coherence in relation to the visual target, jointly affect the extent to which a sound can enhance visual contrast sensitivity. Visual contrast sensitivity was measured by a psychophysical task, where human adult participants reported the location of a visual Gabor pattern presented at various contrast levels. We expected the most enhanced contrast sensitivity, the lowest contrast threshold, when the visual stimulus was accompanied by a task-irrelevant sound, weak in auditory salience, modulated in-phase with the visual stimulus (strong temporal phase coherence). Our expectations were confirmed, but only if we accounted for individual differences in optimal auditory salience level to induce maximal multisensory enhancement effects. Our findings highlight the importance of interactions between temporal phase coherence and stimulus effectiveness in determining the strength of multisensory enhancement of visual contrast as well as highlighting the importance of accounting for individual differences.

Thermal-Tactile Integration in Object Temperature Perception.

The brain consistently faces a challenge of whether and how to combine the available information sources to estimate the properties of an object explored by hand. While object perception is an inference process involving multisensory inputs, thermal referral (TR) is an illusion demonstrating how the interaction between thermal and tactile systems can lead to deviations from physical reality-when observers touch three stimulators simultaneously with the middle three fingers of one hand but only the outer two stimulators are heated (or cooled), thermal uniformity is perceived across three fingers. Here, we used TR of warmth to examine the thermal-tactile interaction in object temperature perception. We show that TR is consistent with precision-weighted averaging of thermal sensation across tactile locations. Furthermore, we show that prolonged contact with TR stimulation results in adaptation to the local variations of veridical temperatures instead of the thermal uniformity perceived across three fingers. Our results illuminate the flexibility of processing that underlies thermal-tactile interactions and serve as a basis for thermal display design.

What makes a shape "baba"? The shape features prioritized in sound-shape correspondence change with development.

Crossmodal sound-shape correspondence, the association of abstract shapes and nonsense words (e.g., "bouba-kiki" effect), is seen across cultures and languages. Recent research suggests that the sensitivity for such sound-shape pairings might increase with development. Here we examined one possible mechanism underlying developmental changes in sensitivity to sound-shape correspondences-if and how shape features, such as contour spikiness and the number and size of protrusions, might be weighted differently by children and adults. In Experiment 1, we asked participants to choose which of two nonsense words matched a given visual shape while manipulating contour spikiness and number and size of protrusions independently. We found that adults associated /i/ sounds with shapes having spiky contours and 3 small protrusions. Of these shape features, contour spikiness showed the strongest association. Whereas 9- to 11-year-olds showed adult-like responses, 6- to 8-year-olds prioritized protrusion number, not contour spikiness. Importantly, in Experiment 2, where contour spikiness was highlighted by presenting round and spikey shapes side by side, 6- to 8-year-olds could make associations based on contour spikiness. Our findings suggest that 6- to 8-year-olds prioritize different features of a shape when making sound-shape correspondence compared with adults. Interestingly, these shape-processing biases can be altered by context such that children can resemble adults when the relevant shape features are highlighted. Our results suggest that biases in visual shape processing and the ability to extract contextual information might be additional factors explaining developmental changes in sensitivity toward sound-shape correspondences. These changing developmental biases highlight the contribution of perceptual processing styles in crossmodal correspondence.

Prefer feeling bad? Subcultural differences in emotional preferences between Han Chinese and Mongolian Chinese.

As a multi-ethnic country that is comprised of diverse cultural systems, there has been little research on the subcultural differences in emotional preferences in China. Also, little attention has been paid to examine how explicit and implicit attitudes towards emotions influence emotional preferences interactively. In this study, we manipulated explicit attitudes towards emotions among Han (N = 62) and Mongolian Chinese individuals (N = 70). We assessed participants' implicit attitudes towards emotions to explore their contributions to emotional preferences. (a) Han Chinese had lower preferences for pleasant emotions than Mongolian Chinese after inducing contra-hedonic attitudes towards emotions, and (b) after priming contra-hedonic attitudes towards emotions, the more Han Chinese participants evaluated pleasant emotions as negative implicitly, the less they preferred to engage in pleasant emotional activities. These findings contribute to the growing literature of subcultural differences and demonstrate that explicit and implicit attitudes towards emotions interactively influence individuals' emotional preferences between different subculture groups.

Preverbal infants utilize cross-modal semantic congruency in artificial grammar acquisition.

Learning in a multisensory world is challenging as the information from different sensory dimensions may be inconsistent and confusing. By adulthood, learners optimally integrate bimodal (e.g. audio-visual, AV) stimulation by both low-level (e.g. temporal synchrony) and high-level (e.g. semantic congruency) properties of the stimuli to boost learning outcomes. However, it is unclear how this capacity emerges and develops. To approach this question, we examined whether preverbal infants were capable of utilizing high-level properties with grammar-like rule acquisition. In three experiments, we habituated pre-linguistic infants with an audio-visual (AV) temporal sequence that resembled a grammar-like rule (A-A-B). We varied the cross-modal semantic congruence of the AV stimuli (Exp 1: congruent syllables/faces; Exp 2: incongruent syllables/shapes; Exp 3: incongruent beeps/faces) while all the other low-level properties (e.g. temporal synchrony, sensory energy) were constant. Eight- to ten-month-old infants only learned the grammar-like rule from AV congruent stimuli pairs (Exp 1), not from incongruent AV pairs (Exp 2, 3). Our results show that similar to adults, preverbal infants' learning is influenced by a high-level multisensory integration gating system, pointing to a perceptual origin of bimodal learning advantage that was not previously acknowledged.

Cross-modal attention influences auditory contrast sensitivity: Decreasing visual load improves auditory thresholds for amplitude- and frequency-modulated sounds.

We used a cross-modal dual task to examine how changing visual-task demands influenced auditory processing, namely auditory thresholds for amplitude- and frequency-modulated sounds. Observers had to attend to two consecutive intervals of sounds and report which interval contained the auditory stimulus that was modulated in amplitude (Experiment 1) or frequency (Experiment 2). During auditory-stimulus presentation, observers simultaneously attended to a rapid sequential visual presentation-two consecutive intervals of streams of visual letters-and had to report which interval contained a particular color (low load, demanding less attentional resources) or, in separate blocks of trials, which interval contained more of a target letter (high load, demanding more attentional resources). We hypothesized that if attention is a shared resource across vision and audition, an easier visual task should free up more attentional resources for auditory processing on an unrelated task, hence improving auditory thresholds. Auditory detection thresholds were lower-that is, auditory sensitivity was improved-for both amplitude- and frequency-modulated sounds when observers engaged in a less demanding (compared to a more demanding) visual task. In accord with previous work, our findings suggest that visual-task demands can influence the processing of auditory information on an unrelated concurrent task, providing support for shared attentional resources. More importantly, our results suggest that attending to information in a different modality, cross-modal attention, can influence basic auditory contrast sensitivity functions, highlighting potential similarities between basic mechanisms for visual and auditory attention.

Eye of origin guides attention away: An ocular singleton column impairs visual search like a collinear column.

Collinearity and eye of origin were recently discovered to guide attention: Target search is impaired if it is overlapping with a collinear structure (Jingling & Tseng, 2013) but enhanced if the target is an ocular singleton (Zhaoping, 2008). Both are proposed to occur in V1, and we study their interaction here. In our 9 × 9 search display (Experiment 1), all columns consisted of horizontal bars except for one randomly selected column that contained orthogonal bars (collinear distractor). All columns were presented to one eye except for a randomly selected column that was presented to the other eye (ocular distractor). The target could be located on a distractor column (collinear congruent [CC]/ocular congruent [OC]) or not (collinear incongruent [CI]/ocular incongruent [OI]). We expected to find the best search performance for OC + CI targets and the worst search performance for OI + CC targets. The other combinations would depend on the relative strength of collinearity and ocular information in guiding attention. As expected, we observed collinear impairment, but surprisingly, we did not observe any search advantage for OC targets. Our subsequent experiments confirmed that OC search impairment also occurred when color-defined columns (Experiment 2), ocular singletons (Experiments 4 and 5), and noncollinear columns (Experiment 5) were used instead of collinear columns. However, the ocular effect disappeared when paired with luminance-defined columns (Experiments 3A and 3B). Although our results agree well with earlier findings that eye-of-origin information guides attention, they highlight that our previous understanding of search advantage by ocular singleton targets might have been oversimplified.

Bimodal emotion congruency is critical to preverbal infants' abstract rule learning.

Extracting general rules from specific examples is important, as we must face the same challenge displayed in various formats. Previous studies have found that bimodal presentation of grammar-like rules (e.g. ABA) enhanced 5-month-olds' capacity to acquire a rule that infants failed to learn when the rule was presented with visual presentation of the shapes alone (circle-triangle-circle) or auditory presentation of the syllables (la-ba-la) alone. However, the mechanisms and constraints for this bimodal learning facilitation are still unknown. In this study, we used audio-visual relation congruency between bimodal stimulation to disentangle possible facilitation sources. We exposed 8- to 10-month-old infants to an AAB sequence consisting of visual faces with affective expressions and/or auditory voices conveying emotions. Our results showed that infants were able to distinguish the learned AAB rule from other novel rules under bimodal stimulation when the affects in audio and visual stimuli were congruently paired (Experiments 1A and 2A). Infants failed to acquire the same rule when audio-visual stimuli were incongruently matched (Experiment 2B) and when only the visual (Experiment 1B) or the audio (Experiment 1C) stimuli were presented. Our results highlight that bimodal facilitation in infant rule learning is not only dependent on better statistical probability and redundant sensory information, but also the relational congruency of audio-visual information. A video abstract of this article can be viewed at https://m.youtube.com/watch?v=KYTyjH1k9RQ.

Invisible collinear structures impair search.

Visual attention and perceptual grouping both help us from being overloaded by the vast amount of information, and attentional search is delayed when a target overlaps with a snake-like collinear distractor (Jingling & Tseng, 2013). We assessed whether awareness of the collinear distractor is required for this modulation. We first identified that visible long (=9 elements), but not short (=3 elements) collinear distractor slowed observers' detection of an overlapping target. Then we masked part of a long distractor (=9 elements) with continuous flashing color patches (=6 elements) so that the combined dichoptic percept to observers' awareness was a short collinear distractor (=3 elements). We found that the invisible collinear parts, like visible ones, can form a continuous contour to impair search, suggesting that conscious awareness is not a pre-requisite for contour integration and its interaction with selective attention.

Collinear integration affects visual search at V1.

Perceptual grouping plays an indispensable role in figure-ground segregation and attention distribution. For example, a column pops out if it contains element bars orthogonal to uniformly oriented element bars. Jingling and Tseng (2013) have reported that contextual grouping in a column matters to visual search behavior: When a column is grouped into a collinear (snakelike) structure, a target positioned on it became harder to detect than on other noncollinear (ladderlike) columns. How and where perceptual grouping interferes with selective attention is still largely unknown. This article contributes to this little-studied area by asking whether collinear contour integration interacts with visual search before or after binocular fusion. We first identified that the previously mentioned search impairment occurs with a distractor of five or nine elements but not one element in a 9 × 9 search display. To pinpoint the site of this effect, we presented the search display with a short collinear bar (one element) to one eye and the extending collinear bars to the other eye, such that when properly fused, the combined binocular collinear length (nine elements) exceeded the critical length. No collinear search impairment was observed, implying that collinear information before binocular fusion shaped participants' search behavior, although contour extension from the other eye after binocular fusion enhanced the effect of collinearity on attention. Our results suggest that attention interacts with perceptual grouping as early as V1.

Falling skyscrapers: when cross-modal perception of verticality fails.

The perception of verticality is critical for balance control and interaction with the world. But this complex process fails badly under certain circumstances-usually as the result of an illusion. Here, we report on a real-world example of how the brain fails to disregard body position on a moving mountain tram and adopts an inappropriate frame of reference, which prompts passengers to perceive skyscrapers leaning by as much as 30°. To elucidate the sensory origin of this misperception, we conducted field experiments on the moving tram to systematically disentangle the contributions of four sensory systems known to affect verticality perception, namely, vestibular, tactile, proprioceptive, and visual cues. Our results refute the intuitive assumption that the perceived tilt of the buildings is based on visual error signals and demonstrate instead that a unified percept of verticality is a product of the synergistic interaction among multiple sensory systems and the contextual information available in the real world.